Wearable robot data collection system with human-machine operation interface

ABSTRACT

A data collection system that performs data collection of human-driven robot actions for robot learning. The data collection system includes: i) a wearable computation subsystem that is worn by a human data collector and that controls the data collection process and ii) a human-machine operation interface subsystem that allows the human data collector to use the human-machine operation interface to operate an attached robotic gripper to perform one or more actions. A user interface subsystem receives instructions from the wearable computation subsystem that direct the human data collector to perform the one or more actions using the human-machine operation interface subsystem. A visual sensing subsystem includes one or more cameras that collect raw visual data related to the pose and movement of the robotic gripper while performing the one or more actions. A data collection subsystem receives collected data related to the one or more actions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalPatent Application No. 63/241,792, filed on Sep. 8, 2021, the entiretyof which is incorporated herein by reference.

FIELD OF THE DISCLOSURE

The disclosure relates to a data collection system that performs datacollection of human driven robot actions for future robot learning.

BACKGROUND

The range of applications of robotic grasping and manipulation has grownover the past decade spanning from industrial automation to applicationsin sectors such as service robotics and household automation. Mostautonomous robots are generally intended and built to be used instructured environments. However, with the recent advancements insensors, actuators, design, and control algorithms, the use ofsimplified actuation schemes, and the incorporation of soft materials inrobot grippers and hands, the shift towards compliant, under-actuateddevices and simplified actuation methods has also allowed forapplications in unstructured and dynamic environments. Such designsincrease the efficiency and robustness of robotic end-effectors inexecuting grasping and manipulation tasks that require significantversatility and dexterity.

Though gripper technologies have advanced significantly, numerousdifficulties associated with the autonomous execution of grasping andmanipulation tasks remain. Significant research effort has been put intothe analysis and synthesis of grasps based on simulations as well asinto the execution of task-oriented grasping using both tactile sensingand vision based methods. Control systems have made use of environmentaland proprioceptive information as well as analytical methods to processthe simulated and real data and generate trajectories for robustgrasping and dexterous manipulation. Though such analytical and hybridapproaches (e.g., physics based simulation) have delivered exceptionalresults in robot grasping and manipulation tasks in structuredenvironments, they often require a highly accurate representation of thereal-world environment that is hard or even infeasible to obtain. Thisfurther increases the complexity of simulations and the requiredcomputational power to generate results. Regarding the availabledesigns, most existing robotic grippers are created to be mounted onrobotic manipulators. They can be interfaced to a programmablecontroller or the manipulator user interface to be programmed for theexecution of specific tasks. However, for the experimental validation ofrobotic gripper and human-machine skill transfer applications, theirperformance is limited by the reachable workspace of the manipulatorsthat they are mounted on. Portable devices and human machine interfacesthat can accommodate and control these grippers allow humans to operatethem more directly, intuitively, and efficiently. Using simple handheldinterfaces, human operators can directly control the motion of therobotic fingers and with additional sensors, capture the motion andmanipulation data required for the execution of complex tasks in bothstructure and unstructured environments. Additionally, having a portableinterface allows grasping and manipulation tasks to be adjusted andprogrammed outside of lab environments in real-world scenarios withsafety. This is particularly useful for testing prototypes in everydaylife settings where autonomous operation of a robot that learns fromsimulated data or experience could pose a health and safety risk. Aportable human-machine operation interface also allows for vigorous andfaster testing of various functionalities and design aspects of roboticgrippers in a more intuitive manner.

Deriving complex in-hand manipulation motions for robotic grippers andhands is an extremely challenging task for both humans and roboticlearning methodologies, as it requires the coordination of multipledegrees of freedom with accuracy and precision. Such a task cannot beaccomplished with traditional control interfaces such as roboticpendants. New more immersive and intuitive interfaces are needed thatoffer more direct control of the robot motion and that allow humans totake full advantage of the available robot dexterity. By providingdirect-mapped controls for human operation, the human talent and acuteproblem-solving capabilities can assist in overcoming the computationalcomplexity of calculating the complex robot kinematic and dynamicbehaviors of dexterous manipulation. Such intuitive interfaces couldalso be utilized to test the full capabilities of the developedprototypes and explore new design opportunities.

Accordingly, there is a need for new, more immersive, and intuitivehuman-machine interfaces that offer more direct control of the robotgripper/hand motions and that allow humans to take full advantage of theavailable robot dexterity. In addition, there is a need for improveddata collection systems to perform data collection of human-driven robotactions for future robot learning.

SUMMARY

Embodiments disclosed herein are related to a data collection systemthat performs data collection of human-driven robot actions for futurerobot learning. The data collection system in one embodiment includesfive subsystems: 1) a visual sensing subsystem, 2) a User Interface (UI)subsystem, 3) a human-machine operation interface subsystem, 4) awearable computation subsystem, and 5) a data store subsystem.

In some embodiments, the visual sensing subsystem includes a camera,such as a depth camera, which is located on the human-machine operationinterface, a bird-view camera, which may also be a depth camera, whichis located on a wall or ceiling of a data collection location, and othercamera and types of sensors as needed. All data collected by the variouselements of the visual sensing subsystem is collected with regularintervals or triggered by an event during data collection. The collecteddata is transferred to the wearable computation subsystem and/or thedata store subsystem.

In some embodiments, the user interface subsystem is a platform thatallows a data collector to communicate with the various other subsystemsof the data collection system. In the embodiments, the user interfacecontains a display monitor, an AR/VR device, a voice user interface,and/or a combination of these devices. The data collector receives andvisualizes via the user interface sensing data, instructions, andfeedback from the wearable computation subsystem and provides commandsvia the user interface to the wearable computation subsystem forrecording.

In some embodiments, the human-machine operation interface is a forearmmounted human-machine operation interface that is used to operate one ormore robotic grippers or hands in the execution of complex grasping andmanipulation tasks. The forearm mounted human-machine operationinterface includes a forearm stabilizer platform that attaches to ahuman data collector's forearm. A gripper support arm has a first endcoupled to an end of the forearm stabilizer platform. A gripper couplingmember is coupled to a second end of the gripper support arm. Thegripper coupling member couples the one or more robotic grippers orhands to the forearm mounted human-machine operation interface so thatthe data collector can operate the one or more robotic grippers or handswith ease. A grip handle is connected to the gripper support arm toprovide extra support. The grip handle accommodates at least one inputinterface that receives user input and provides appropriate controlcommands to a microcontroller unit to control an operation of the one ormore robotic grippers or hands. The forearm mounted human-machineoperation interface and/or the one or more robotic grippers and handsinclude various sensors and control signals that are used for datacollection. The collected data is provided to the wearable computationsubsystem for recording.

In other embodiments, the human-machine operation interface is a palmmounted human-machine operation interface that is used to operate one ormore robotic grippers or hands in the execution of complex grasping andmanipulation tasks. The palm mounted human-machine operation interfaceincludes an interface body and a palm support coupled with the interfacebody. A gripper coupling member is coupled to the interface body. Thegripper coupling member connects the one or more robotic grippers orhands to the palm mounted human-machine operation interface so that thedata collector can operate the one or more robotic grippers or hands.The palm mounted human-machine operation interface includes at least oneinput interface that receives user input and provides appropriatecontrol commands to a microcontroller unit to control the operation ofthe one or more robotic grippers or hands. The palm mountedhuman-machine operation interface and/or the one or more roboticgrippers and hands include various sensors and control signals that areused for data collection. The collected data is provided to the wearablecomputation subsystem for recording.

In some embodiments, the wearable computation subsystem overseesreal-time synchronization of multiple data resources, data processing,and data visualization by providing commands to and receiving collecteddata from the visual sensing subsystem, the user interface, and thehuman-machine operation interface. The collected data is sent to thedata store subsystem.

In some embodiments, the data store subsystem includes a visual datastore and a wearable data store. In operation, all the collected dataincluding sensing data and control signal data is stored in the datastore subsystem. The collected data provided by the wearable computationsubsystem will be stored in the wearable data store. Data collected bythe bird-view camera will be stored in the visual data store. This useof the visual data store is typically useful for the bird-view cameraswhich cannot connect to the wearable computation subsystem withoutsacrificing mobility.

These and other features, aspects, and advantages of the presentdisclosure will become better understood through the followingdescription, appended claims, and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate a data collection system that performs datacollection of human-driven robot actions for future robot learningaccording to embodiments disclosed herein.

FIGS. 2A and 2B illustrate a data collection environment according toembodiments disclosed herein.

FIGS. 3A and 3B illustrate a forearm mounted human-machine operationinterface according to embodiments disclosed herein.

FIG. 4 illustrates various robotic grippers or hands that may beoperated by the forearm mounted human-machine operation interface and/orthe palm mounted human-machine operation interface disclosed herein.

FIG. 5 illustrates aspects of the forearm mounted human-machineoperation interface and the palm mounted human-machine operationinterface disclosed herein.

FIGS. 6A and 6B illustrate a palm mounted human-machine operationinterface according to embodiments disclosed herein.

The drawing figures are not necessarily drawn to scale. Instead, theyare drawn to provide a better understanding of the components thereof,and are not intended to be limiting in scope, but to provide exemplaryillustrations.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

A better understanding of the disclosure's different embodiments may behad from the following description read with the drawings in which likereference characters refer to like elements. While the disclosure issusceptible to various modifications and alternative constructions,certain illustrative embodiments are shown in the drawings and aredescribed below. It should be understood, however, there is no intentionto limit the disclosure to the specific embodiments disclosed, but onthe contrary, the aim is to cover all modifications, alternativeconstructions, combinations, and equivalents falling within the spiritand scope of the disclosure.

The references used are provided merely for convenience and hence do notdefine the sphere of protection or the embodiments. It will beunderstood that unless a term is expressly defined in this applicationto possess a described meaning, there is no intent to limit the meaningof such term, either expressly or indirectly, beyond its plain orordinary meaning. Any element in a claim that does not explicitly state“means for” performing a specified function or “step for” performing aspecific function is not to be interpreted as a “means” or “step” clauseas specified in 35 U.S.C. § 112.

FIGS. 1A and 1B illustrate an example embodiment of a data collectionsystem 100 that performs data collection of human-driven robot actionsfor future robot learning. As illustrated, the data collection system100 includes various subsystems that are configured to be used by a datacollector 105 when collecting data related to human-driven robot actionsfor future robot learning. The data collector 105 will typically be ahuman who wears at least some of the various robotic subsystems whilecollecting the data as will be explained in more detail to follow. Thewearability of the data collection system 100 (or at least some of thesubsystems) allows the data collector 105 to mimick actual human actionor movement using the robotic end-effector and this can then be recordedand used as data for future robot learning.

As illustrated, the data collection system 100 includes a wearablecomputation subsystem 110. As illustrated in FIG. 1B, the wearablecomputation subsystem includes a wearable computation and data storagedevice 115, which in the embodiments may comprise any reasonablecomputing system. As illustrated, the wearable computation and datastorage device 115 is mounted onto a frame 116 and the frame 116 isconnected to an arm member 117 that is configured to be attached to awearable rig vest that will be described in more detail to follow.Advantageously, the frame 116 and the arm member 117 allow the wearablecomputation and data storage device 115 to be mounted onto the back ofthe data collector 105 so as to not impede mobility of the datacollector when performing the data collection. In other embodiments, therig vest may include a backpack like structure that allows the wearablecomputation and data storage device 115 to be directly mounted onto therig vest so as to be mounted onto the back of the data collector 105without the need for the frame 116.

The wearable computation subsystem 110 is the centre of the datacollection system 100. In some embodiments, the wearable computationsubsystem performs at least the following four functions: 1) real-timesynchronization of multiple data resources, 2) data processing, 3)creating data visualization, and 4) communication with a user interface(UI) subsystem 120. The wearable computation subsystem 110 providestimestamps for each piece of incoming data. The timestamps are usefulfor future time synchronization. The wearable computation subsystem 110is configured to perform necessary and simple data processing, such asresampling, resizing images, and pose estimation. The processed data isthen transferred to a data store subsystem 150. The wearable computationsubsystem 110 is also configured to perform simple graphic plotting forreal time data visualization. The wearable computation subsystem 110provides essential information, e.g., pose estimation, target selection,instructions for the UI subsystem 120 to respond to the data collector105. In addition, the wearable computation subsystem 110 also receivesand puts timestamps of the data collector's commands, feedback, andannotations and then transfers this data to the data store subsystem150. The interactions of the wearable computation subsystem 110 with thesubsystems of the data collection system 100 will be described in moredetail to follow.

As illustrated, the data collection system 100 also includes the UserInterface (UI) subsystem 120 (also referred to herein as “UI 120”). Inembodiments, the UI 120 is a platform that the data collector 105 canuse to communicate with the other subsystems of the data collectionsystem 100. The UI 120 is a bidirectional interface. In one side the UI120 delivers sound and visual information including instructions,confirmations, and previews from the other subsystems of the datacollection system 100 to the data collector 105. In some embodiments,the UI 120 may include a display monitor, a VR/AR device that may showcamera previews, data curves, and visual demos/instructions or voiceprompts. In the other side, the UI 120 collects comments, feedbacks, andcommands from the data collector 105. The UI 120 may involve a voiceuser interface to listen to the data collector's command or motioncommand interface through the VR/AR device or other hardware inputdevices like buttons, keyboard, or a mouse.

As mentioned previously, in some embodiments, the UI 120 includes aVirtual Reality/Augmented Reality (VR/AR) device such as VR/AR glasses121 and a voice user interface 122. The VR/AR glasses 121 may be anyreasonable type of VR/AR glasses that can be used to provideinstructions to the data collector 105. The voice user interface 122 maybe implemented as earphones or speakers that are part of the VR/ARglasses 121 or that may be separate hardware from the VR/AR glasses 121.In addition, the voice user interface 122 may include a microphone thatallows the data collector 105 to provide voice commands and feedback asneeded.

In operation, the VR/AR glasses 121 and/or the voice user interface 122allow the data collector 105 to receive instructions 111 from thewearable computation subsystem 110. For example, in one embodiment thevoice user interface 122 allows the user to hear voice instructions asan example of the instructions 111 from the wearable computationsubsystem 110. The voice instructions may audibly instruct the datacollector 105 to pick up an object of interest such as a cup with ahuman-machine operation interface 130 and connected robotic gripper 170and then to move the cup so that the data related to this movement canbe collected.

In some embodiments, the instructions 111 may be implemented as datavisualization instructions that provide visual instructions to the datacollector 105. That is, the visual instructions are generated by theVR/AR glasses 121 so as to be viewable by the data collector 105. Forexample, FIG. 2A illustrates a data collection environment 200 includinga cup 201 and a cup 202. In this embodiment, a data visualizationinstruction is implemented as a bounding box 210 that surrounds the cup201, thus informing the data collector 105 that he or she should pick upthe cup 201 with the human-machine operation interface 130 and connectedrobotic gripper 170 and not the cup 202. In addition, the datavisualization instructions 111 may also include readable instructionsthat can be read by the data collector 105. As illustrated, the readableinstructions 211 may instruct the data collector to “pick up the cup”and the readable instructions 212 may instruct the data collector to“move the cup 10 inches to the right”. It will be appreciated that theremay be any number of readable instructions as circumstances warrant. Insome embodiments, a demonstration of how to perform the instructedaction such as picking up and moving the cup 201 may be shown to thedata collector 105 in the VR/AR glasses 121. This helps ensure that thedata collector 105 will be properly trained to carry out the instructedaction.

In operation, the VR/AR glasses 121 and/or the voice user interface 122also allow the data collector 105 to receive feedback 112 from thewearable computation subsystem 110. For example, as shown in FIG. 2B,the data collector 105 receives status information 220, which is a typeof feedback 112, about the data collection process from the wearablecomputation subsystem 110. As illustrated, the status information 220includes hardware status information 221 that provides information tothe data collector 105 about whether the hardware of the varioussubsystems of the data collection system 100, including the sensors ofthe human-machine operation interface 130 and attached robotic gripper170, are properly functioning. For example, suppose that in one instanceof data collection, one or more of the sensors of the human-machineoperation interface 130 and attached robotic gripper 170 were notfunctioning properly. This would lead to either no data being collectedor being improperly collected. If the data collector 105 is not madeaware that the one or more sensors are not functioning properly, then heor she could waste valuable time collecting erroneous data. By informingthe data collector 105 of a potential hardware problem, remedial actioncan be quickly taken and erroneous data collection can be avoided.

The status information 220 also includes data collection statusinformation 222 that provides information to the data collector aboutthe status of the data collection process and how the process isproceeding to completion. For example, the data collection statusinformation 222 may include a checklist of actions that need to becompleted in order to perform the data collection task. In someembodiments, the data collection status information 222 may also includeprompts to the data collector 105 when an action needs to be repeated. Aprompt may also be provided to the data collector 105 when he or she hasspent a lot of time between actions so as to instruct the data collectorto once again resume the data collection process.

The status information 220 also includes information 223 that informsthe data collector that the data collection process has beensuccessfully completed. It will be appreciated that there may be anynumber of additional status information as circumstances warrant.Although the status information 220 is shown as being generated by theVR/AR glasses 121 to be readable by the data collector 105, in someembodiments the status information 220 may be audibly provided to thedata collector 105 via the voice user interface 122.

In some embodiments, the feedback 112 may also include visualizationinformation. For example, as the data collector 105 follows theinstructions to move the cup 201 using the human-machine operationinterface 130 and attached robotic gripper 170, the bounding box 210 maymove with the cup so as to continually surround the cup while the cup isbeing moved.

The UI 120 also allows the data collector 105 to provide commands 113 tothe wearable computation subsystem 110. For example, in one embodimentthe data collector 105 may use the voice user interface 122 to provideaudio commands such as “begin recording” at the start of a datacollection process or at the start of an instructed action or “stoprecording” at the end of the data collection process or the end of aninstructed action. In another embodiment, the data collector can make ahand gesture that is tracked by the VR/AR glasses 121 to start and stoprecording at the start and completion of the data collection processand/or the start and completion of an instructed action. In still otherembodiments, a QR code that is located in the data collection locationand that can be scanned by the VR/AR glasses 121 can be provided tostart and stop recording at the start and completion of the datacollection process and/or the start and completion of an instructedaction. In further embodiments, a button or other user interface elementcan be included on the human-machine operation interface 130 that ispushed to start and stop recording at the start and completion of thedata collection process and/or the start and completion of an instructedaction. It will be appreciated that the command 113 may provide othertypes of feedback to the wearable computation subsystem 110 in additionto the command to start and stop recording. It will also be appreciatedthat the various implementations of the command 113 allows the datacollector 105 to communicate and provide feedback to the wearablecomputation subsystem 110 without actually touching a button or the likeon the wearable computation subsystem 110. As discussed above, thewearable computation subsystem 110 is worn on the back of the datacollector 105 to ensure the wearable computation subsystem 110 does notimpede the human-machine operation interface 130 and attached roboticgripper 170 during the data collection process.

As illustrated, the data collection system 100 includes thehuman-machine operation interface 130. The human-machine operationinterface 130 includes various gripper sensors 131, some of which areconnected to the human-machine operation interface 130 and some of whichare connected to the attached robotic gripper 170. The gripper sensors131 may include distance sensors and localization sensors that measurethe movement of the human-machine operation interface 130 and/or therobotic gripper 170. The gripper sensors 131 may measure a change inpeak current of one or more motors that are configured to move thefingers of the robotic gripper 170. The data that is measured by thevarious gripper sensors 131 is then sent as human control data 133 tothe wearable computation sub system 110.

The human-machine operation interface 130 may also measure controlsignals 132 that are used to control the various operational elements ofthe human-machine operation interface. For example, the human-machineoperation interface 130 may include one or more buttons, triggers,potentiometers, or other user input devices that are activated by thedata collector 105 when he or she performs an instructed action or task.The activation by the data collector 105 may generate the controlsignals 132. In one embodiment, a control signals 132 may be thepressure that is exerted when the button is activated. The controlsignals 132 are then sent as human control data 133 to the wearablecomputation sub system 110.

As illustrated, the data collection system 100 includes the visualsensing subsystem 140. The visual sensing subsystem 140 includes camerasfor collecting raw sensing camera data. The cameras include depthcameras, tracking cameras, and RGB cameras on the human-machineoperation interface and/or fixed-locations like bird-view cameras. Inone embodiment, one or more bird-view cameras 141 are implemented in afixed location on a wall or ceiling of the location where the datacollection process is occurring. The one or more bird-view cameras canbe a tracking camera such as an Intel T265 camera. In operation, the oneor more bird-view cameras 141 may provide the raw sensing camera data143 to a visual data store 151 of the data store subsystem 150 and/ormay provide raw sensing camera data 144 to the wearable computationsubsystem 110.

In other embodiments, the visual sensing subsystem 140 also includesvarious cameras 142 that are included on the human-machine operationinterface 130. In one embodiment, the various cameras 142 include atleast one depth camera 142A such as an Intel D435 that is mounted on atop surface of the human-machine operation interface 130 and at leastone tracking camera 142B such as an Intel T265 that is mounted on abottom side of the human-machine operation interface 130. These camerasare mounted onto the human-machine operation interface 130 as will beexplained in more detail to follow. In operation, the one or morecameras 142 may provide the raw sensing camera data 143 to the visualdata store 151 of the data store subsystem 150 and/or may provide rawsensing camera data 144 to the wearable computation subsystem 110.

In operation, the raw sensing camera data 143 and/or 144 may includeposition (RGB and depth) data of the human-machine operation interface130 and its related robotic gripper 170 when the data collector 105moves the machine operation interface 130 and its related roboticgripper 170 while perfuming one or more of the instructed actions ortasks. This data is collected by the at least one bird-eye view depthcamera and/or the depth camera implemented on the human-machineoperation interface 130. This data may be collected as video data andcollected on a frame-by-frame basis.

The raw sensing camera data 143 and/or 144 may also include pose datathat tracks a pose of the robotic gripper 170 while perfuming one ormore of the instructed actions or tasks. This data is collected by thebird-eye tracking camera and/or the tracking camera implemented on thehuman-machine operation interface 130. The pose data may include thepose of the gripper 170 in 3D space including X,Y,Z coordinates andtheir related angles.

As illustrated, the data collection system 100 includes the data storesubsystem 150. The data store subsystem 150 may be implemented as partof the wearable computation subsystem 110 or it may be implemented as aseparate computing system. In one embodiment, the data store subsystem150 may be partially implemented on the wearable computation subsystemand partially implemented on the separate computing system. For example,the visual data store 151 may be implemented on the separate computingsystem and the wearable data store 152 may be implemented on thewearable computation subsystem 110. This is advantageous as for mobilityreasons there may be no easy way to connect the visual sensing subsystemto the wearable computation subsystem. As previously described, all datathat is collected by the various subsystems of the data collectionsystem 100 are sent to the data store subsystem 150 for storage. Asdescribed, the data collected by the visual sensing subsystem 140 isstored in the visual data store 151 and the data that is collected bythe wearable computation subsystem 110 is stored in the wearable datastore 152.

As illustrated, the data stored in the data store subsystem 150undergoes a synchronization step 153 that synchronizes the data storedin the visual data store 151 to the data stored in the wearable datastore 152 after the data collection process is completed to generate acomplete final synchronized dataset 160. The complete final synchronizeddataset 160 may then be used in a future process to train one or morerobots to mimic human behaviour.

Specific embodiments of the human-machine operation interface 130 andthe attached robotic gripper 170 will now be explained. FIGS. 3A and 3Billustrate an embodiment of a forearm mounted human-machine operationinterface 300, which corresponds to the human-machine operationinterface 130. These figures also show components configured to beattached to the forearm mounted human-machine operation interface 300 orto help stabilize the interface as will be explained. The forearmmounted human-machine operation interface 300 is configured to assist auser while operating a robotic gripper or hand 400 and to help reduce auser's arm fatigue while operating a robotic gripper or hand 400. Therobotic gripper or hand 400 corresponds to the robotic gripper 170. Theforearm mounted human-machine operation interface 300 creates spacearound a user's hand so as to facilitate the operation of the interfaceusing the included input interfaces while the forearm mountedhuman-machine operation interface 300 is resting on a user's forearm.

As shown in FIGS. 3A and 3B, the forearm mounted human-machine operationinterface 300 may include a forearm stabilizer platform 310, which maybe constructed of any reasonable material such as a metal or a plastic.In the embodiment, the forearm stabilizer platform 310 may include arectangular shaped platform portion 312 that is configured to support anelectronics circuit board such as the electronics circuit board 324. Theforearm stabilizer platform 310 may also include a connection portion313 that is located on one end of the rectangular shaped platformportion 312. The connection portion 313 includes connection holes 314,315, 316, and 317 that are used to connect the forearm stabilizerplatform 310 to a gripper support arm 330 as will be explained in detailto follow.

The forearm stabilizer platform 310 may also include a first downwardextending portion 311A and a second downward extending portion 311B thatextend downward from the rectangular shaped platform portion 312.Although not illustrated due to the orientation of FIGS. 3A and 3B, theforearm stabilizer platform 310 may also include a third and fourthdownward extending portions that are on opposite sides of therectangular shaped platform portion 312 from the downward extendingportions 311A and 311B and that also extend downward from therectangular shaped platform portion 312. The four downward extendingportions are shaped so as to conform to and rest on a user's forearmwhen in use. As shown in FIG. 3B, in some embodiments padding 328, whichmay be any reasonable padding such as a cloth sponge, may be attached tothe four downward extending portions to increase comfort for a user whowears the forearm mounted human-machine operation interface 300.

The rectangular shaped platform portion 312 may include a first opening312A that has a first slot 312A1 and a second, non-illustrated slot thatis on an opposite side to the first slot 312A1. The rectangular shapedplatform portion 312 may also include a second opening 312B that has afirst slot 312B1 and a second, non-illustrated slot that is on anopposite side to the first slot 312B1. As shown in the figures, thevarious slots may be located adjacent the downward extending portions311A and 311B. As shown in FIG. 3B, the first and second openings 312Aand 312B and their associated slots may be used to receive fasteningstraps 327 that may be made of Velcro or any other reasonable material,and that are configured to secure the forearm mounted human-machineoperation interface 300 to the forearm of a user so that the interfaceis properly supported on the forearm of the user/operator when in use.

As discussed previously, the electronics circuit board 324 may be placedon the top side of the rectangular shaped platform portion 312 such thatthe electronics circuit board 324 is supported by the rectangular shapedplatform portion 312. To protect the electronics circuit board, in someembodiments the forearm mounted human-machine operation interface 300may include electronics cover or housing 320, which may be constructedof any reasonable material such as a metal or a plastic. The electronicscover or housing 320 may include a central portion 322 and a firstsidewall 321A and a second sidewall 321B that both extend from thecentral portion.

As shown in FIG. 3B, the electronics cover or housing 320 is configuredso that the sidewalls 321A and 321B fit around and are secured to therectangular shaped platform portion 312 between the extending portions311 to thereby protect the electronics circuit board 324 when present.As further illustrated, the electronics cover or housing 320 may includeindicator LEDs 323A that are configured to connect with some of theelectronics on the electronics circuit board 324 and light-up when theelectronics are being used or otherwise supplied with a power source toindicate the operation state or other important information, pertainingto the operation of the device. In addition, electronics cover orhousing 320 may also include an opening 323B that is configured to allowvarious electrical wires to connect to the electronic components of theelectronics circuit board 324. In some embodiments, the electronicscover or housing 320 may not cover an interface portion 325 of theelectronics circuit board 324 so as to allow various electrical wiring326 to run from the electronics circuit board 324 and other elements ofthe forearm mounted human-machine operation interface 300. It will beappreciated that the electronics circuit board 324 may include varioustypes of electronics such as a computing unit, control circuitry, andthe like that can be used to control the various elements of the forearmmounted human-machine operation interface 300 and the robotic gripper orhand 400 that is attached to the human-machine operation interface 300.In addition, the various electronics can be used to collect and plot inreal-time data about the usage of the forearm mounted human-machineoperation interface 300 and the connected robotic gripper or hand 400.

As shown in FIGS. 3A and 3B, the forearm mounted human-machine operationinterface 300 may include a gripper support arm 330, which may beconstructed of any reasonable material such as a metal or a plastic. Inthe embodiment, the gripper support arm 330 includes a central portion331 that may have a curved shape and a connection portion 332 located ata first end of the central portion 331. The connection portion 332includes a first connection hole 333, a second connection hole 334, athird connection hole 335 that is on the opposite side of the connectionportion from the first connection hole 333, and a fourth connection holethat is on the opposite side of the connection portion from the secondconnection hole 334.

In this embodiment, the connection portion 332 cooperates with theconnection portion 313 to connect the gripper support arm 330 and theforearm stabilizer platform 310. Specifically, the connection portion332 is inserted into the connection portion 313 so that the connectionholes 316 and 334 align, the connection holes 317 and 335 align, theconnection holes 314 and 333 align, and the connection holes 315 and 336align. A first quick slot peg or some other suitable connection member344 is configured to be inserted into the connection channel created bythe alignment of the connection holes 316, 317, 334, and 335 to therebysecure the gripper support arm 330 to the forearm stabilizer platform310. A second quick slot peg or some other suitable connection member345 is configured to be inserted into the connection channel created bythe alignment of the connection holes 314, 315, 333, and 336 to furthersecure the gripper support arm 330 to the forearm stabilizer platform310. The second quick slot peg 345 is also configured to function as alocking mechanism so that removal of the second quick slot peg 345allows for the execution of wrist flexion/extension motion.

In the embodiment, the gripper support arm 330 includes a mount portion337 that is located on a second end of the central portion 331. Themount portion 337 includes various mounting holes 338, 339, 340, 341,and 342 that are configured to mount various other elements to thegripper support arm 330 as will be explained.

As shown in FIGS. 3A and 3B, the forearm mounted human-machine operationinterface 300 may include grip handle 350. The grip handle 350 mayinclude a central grip portion 351 that includes an ergonomic grip forproviding comfort to a user's fingers when using the grip handle 350 tocontrol the forearm mounted human-machine operation interface 300. Thegrip handle 350 also includes a first side portion 352 and a second sideportion 353 that are connected to the central grip portion 351. Anextending portion 354 extends from the second side portion 353 and isused to help connect the grip handle 350 to the gripper support arm 330.The connection between the grip handle and the support arm is achievedusing a bolt and nut on either side of the gripper support arm panelalong a swivel axis. The grip handle 350 connector has a slit to allowadjustment of the handle position. A secondary bolt and nut can beinstalled in the hole 342 to lock the grip handle 350 in position.Otherwise, the grip handle 350 can be swiveled around the swivel axis byloosening the connecting bolt and nuts.

In the embodiment, the grip handle 350 is also configured to haveseveral input interfaces mounted thereon. For example, a push button355, which may be a 6 mm push button, may be mounted onto the first sideportion 352 using any reasonable attachment mechanism. In addition, arotary potentiometer 356 may also be mounted onto the first side portion352. The rotary potentiometer 356 may include a shaft or rotation memberthat is configured to be connected to a trigger member 357 via a hole ofthe trigger member. The trigger member 357 allows the user to adjust thepotentiometer as needed when controlling the attached robotic gripper orhand 400. In some embodiments, the trigger member 357 may be replacedwith a slider control or other suitable member that is configured toadjust the rotary potentiometer 356. The rotary potentiometer 356 andtrigger member 357 may be attached to the grip handle 350 using anyreasonable fasteners such as the screws 358 and nuts 359 shown in FIGS.3A and 3B.

In the embodiment, a 12-pulse incremental encoder 360 may also bemounted onto the first side portion 352. The incremental encoder 360 mayinclude an encoder wheel shaft 361 that connects a scroll wheel 362 tothe incremental encoder 360. The scroll wheel 362 allows the user tocontrol the incremental encoder 360 as needed when controlling theattached robotic gripper or hand 400. The incremental encoder 360 andthe scroll wheel 362 may be attached to the grip handle 350 using anyreasonable fasteners such as the screws 358 and nuts 359 shown in FIGS.3A and 3B.

As shown in FIGS. 3A and 3B, the forearm mounted human-machine operationinterface 300 may include a universal joint adaptor 380 that isconfigured to mount a universal joint 385 to the gripper support arm330. The universal joint adaptor 380 includes connection holes that areused to mount the adaptor to the gripper support arm 330 using anyreasonable fasteners. The universal joint 385 is configured to allow forthe forearm mounted human-machine operation interface 300 to couple withan iso-elastic arm 510 of a wearable vest rig 500 that is designed toprovide further support to the forearm mounted human-machine operationinterface 300 and thus function as a support structure as will beexplained in more detail to follow.

As shown in FIGS. 3A and 3B, the forearm mounted human-machine operationinterface 300 may include a camera mount adaptor 370 that is configuredto mount one or more cameras, such as the camera 395 that in mounted ona top surface of the camera mount adaptor 370. Although not illustrated,a second camera may be mounted on a bottom surface of the camera mountadaptor 370. The camera mount adaptor 370 may include a first arm 371A,a second arm 371B, a third arm 371C, and a fourth arm 371D or even morearms. Each of the arms 371 may include a corresponding hole that may beused to mount a camera such as the camera 395. In the embodiment, thefour arms 371 are able to accommodate one camera as shown in FIG. 3B orthey may accommodate two cameras in a V shape. It will be appreciatedthat in other embodiments the camera mount adaptor 370 may be of otherreasonable shapes that are configured to host different angles ornumbers of cameras. The camera 395 is used as a vision capture devicefor observing the operation of the attached robotic gripper or hand 400.The camera 395 may be attached to the holes of one or more of the arms371A-371D by any reasonable fasteners such as the screws 358 and nuts359. In some embodiments, a camera offset mount 396 may be implementedin place of or in conjunction with the camera mount adaptor 370. Thecamera offset mount 396 may attach to the camera mount adaptor 370 ordirectly to the gripper support arm 330. The camera offset mount 396 maybe secured to the camera and the camera mount adaptor 370 by sets ofembedded nuts and bolts. The camera offset mount 396 has nut housingcavities for connections with the camera mount adaptor 370 and thinthrough hole plates for direct bolt and screw mounting to the body ofcameras 395. This design allows the use of the same screws to be usedwhen mounting the camera with and without the camera offset mount 396.

The camera mount adaptor 370 also includes various connection holes372-378 that are used to connect the camera mount adaptor 370 to otherelements of the forearm mounted human-machine operation interface 300.For example, the connection holes 372-378 may couple with the connectionholes 338-341 of the mount portion 337 to thereby mount the camera mountadaptor 370 to the gripper support arm 330 using any reasonablefasteners such as the screws 358 and nuts 359.

In the embodiment, a gripper coupling member 390 is configured to attachto a side of the camera mount adaptor 370 that is opposite of the sidethat attaches to the mount portion 337. The gripper coupling member 390attaches to the camera mount adaptor 370 using one or more of theconnection holes 372-378 that couple with various connection holes ofthe gripper coupling member 390 that are not shown in the figures.

The gripper coupling member 390 includes a coupling portion 391 that isconfigured to couple with a coupling portion of one or more roboticgrippers or hands 400. It will be appreciated that in some embodiments,a particular gripper coupling member 390 may be configured for aparticular type or model of robotic grippers or hands 400. Thus, in suchembodiments, one gripper coupling member 390 may be exchanged foranother gripper coupling member 390 when a different type or model ofrobotic grippers or hands 400 is to be used. Thus, the forearm mountedhuman-machine operation interface 300 is not limited to operating onlyone type or model of robotic grippers or hands 400. Rather, the forearmmounted human-machine operation interface 300 is configured to operate avariety of robotic grippers or hands 400 using an appropriate grippercoupling member 390.

FIG. 4 illustrates some example robotic grippers or hands 400 that maybe operated by the forearm mounted human-machine operation interface 300or a palm mounted human-machine operation interface 600 to be describedlater. As illustrated, the robotic gripper or hand 400 may be ananthropomorphic gripper 401A or 401B that include five fingers that aresimilar to those of a human hand and that have various different jointsthat are bendable according to commands received from the human-machineoperation interface. The robotic gripper or hand 400 may be amulti-modal parallel gripper 402A (as is also illustrated in FIG. 3B) or402B that has two moving fingers each having various different jointsthat are bendable according to commands received from the human-machineoperation interface. The robotic gripper or hand 400 may also be an openhanded gripper 403 with three fingers. The robotic gripper or hand 400may be an open handed gripper 404 with two fingers. It will beappreciated that the robotic grippers or hands 400 shown in FIG. 4 areexamples only and that other types of reasonable robotic grippers orhands 400 may also be operated by the forearm mounted human-machineoperation interface 300 or palm mounted human-machine operationinterface 600.

FIG. 5 illustrates an embodiment of the vest rig 500 that is configuredto support the forearm mounted human-machine operation interface 300.The vest rig 500 includes a garment 501 designed to be worn by the user.The garment 501 includes various buckles 502 that are configured tosecure the vest rig to the user. Although not illustrated, the back ofthe vest rig 500 may provide mounting space for one or more of powerelectronics, external data storage, step-up converters for added powerfor the various robotic grippers or hands 400, and an on-board computer,such as the wearable computation and data storage device 115 of thewearable computation subsystem 110.

The vest rig 500 may include a rig frontal mounting plate 520 that isconfigured to attach to the garment 501. The rig frontal mounting plate520 may include various connection holes 521 that are used to attach therig frontal mounting plate 520 to the garment 501 via buckles 502 asshown in FIG. 5 . As shown in FIG. 3A, the rig frontal mounting plate520 is formed of a Y shaped upper portion 523, a mid-section plateportion 524 that connects to upper portion 523, and a waist portion 525that connects to the plate portion 524 and is positioned at the waist ofthe vest rig 500. The waist portion 525 includes an arm mounting block530 for mounting a first end of an extender arm 535 that extends fromthe waist portion 525 and has a shaft 536 on a second end for mountingthe iso-elastic arm 510.

In the embodiment, the iso-elastic arm 510 may be mounted from the waistportion 525 of the vest rig 500 and is configured to provide support forthe weight of the forearm mounted human-machine operation interface 300and the attached robotic gripper or hand 400 to thereby reduce userfatigue when operating the robotic gripper or hand 400. The iso-elasticarm 510 may comprise a first horizontal bar portion 511 and a secondhorizontal bar portion 512 that extend out from the vest rig 500 andthat are separated by a first vertical bar portion 513 and a secondvertical bar portion 514 to thereby form a space between the first andsecond bar horizontal portions as shown in FIG. 1A. The iso-elastic arm510 includes a mounting portion 515 that includes a hole that isconfigured to couple to the shaft 536 of the extender arm 535 to therebymount the iso-elastic arm 510 to the waist portion 525.

The iso-elastic arm 510 may also include a mounting pin 516 that extendsupward from the first horizontal bar portion 511. The mounting pin 516is configured to couple with universal joint 385 of the forearm mountedhuman-machine operation interface 300. This allows for the forearmmounted human-machine operation interface 300 to be removed as neededfrom the vest rig 500 so that actions such as changing from one type ofrobotic gripper or hand 400 to another type can be performed.

FIGS. 6A and 6B illustrate an embodiment of a palm mounted human-machineoperation interface 600, which corresponds to the human-machineoperation interface 130. These figures also show components configuredto be attached to the palm mounted human-machine operation interface 600or to help stabilize the interface as will be explained. The palmmounted human-machine operation interface 600 is configured to assist auser while operating a robotic gripper or hand 400 while using the palmof the hand to constrain the system to the hand of the user whileoperating a robotic gripper or hand 400. The palm mounted human-machineoperation interface 600 allows the user to hold the device withoutrequiring the user's fingers to provide support, thereby leaving theuser's fingers free to operate various input interfaces to control theoperation of the robotic gripper or hand 400.

As shown in FIGS. 6A and 6B, the palm mounted human-machine operationinterface 600 may include an interface body 610, which may beconstructed of any reasonable material such as a metal or a plastic. Inthe embodiment, the interface body 610 may include a palm support mountportion 611 that is located on one end of the interface body. The palmsupport mount portion 611 may include machined ridge 612 that is shapedto receive a palm support 640. The palm support mount portion 611 mayinclude components that are configured to secure the palm support 640 tothe interface body 610.

The palm support 640 may include a raised central portion that isdesigned to be ergonomically comfortable to the user when operating therobotic gripper or hand 400. Thus, the palm support 640 may beconstructed of any reasonable material that is able to conform to thehuman palm in a comfortable manner while also providing sufficientsupport. The palm support 640 also includes a ring portion that isconfigured to couple with the machined ridge 612 so that the palmsupport 640 is securely mounted onto the interface body 610.

In the embodiment, the interface body 610 is coupled to a first elasticwrist strap mount 621 and a second wrist mount 622 that extend upwardfrom the interface body 610 and are configured to mount an elastic wriststrap 630. As shown in FIG. 6A, the first elastic wrist strap mount 621is connected to the interface body mount by fasteners 621A and 621Bwhich are inserted into corresponding connection holes of both the firstelastic wrist strap mount 621 and the interface body 610. Although notshown in the figures, the second elastic wrist strap mount 622 isconnected to the interface body mount by fasteners which are insertedinto corresponding connection holes of both the second elastic wriststrap mount 622 and the interface body 610. The second elastic wriststrap mount 622 includes connection holes 622A and 622B that areconfigured to couple with connection holes of the elastic wrist strap630 to secure elastic wrist strap to the second elastic wrist strapmount 622. Although not illustrated, the first elastic wrist strap mount621 has corresponding connection holes for coupling to the elastic wriststrap 630.

The elastic wrist strap 630 may be made of any reasonable elasticmaterial and is configured to go over the backside of a user's hand asshown in FIG. 5 to secure the palm of the user to the palm mountedhuman-machine operation interface 600 when operating the robotic gripperor hand 400. The elastic wrist strap 630 may include a central portion631 that goes over the backside of the user's hand. A first arm 632 andsecond arm 633 may extend from the central portion 631 and may includeconnection holes 636 and 637 respectively for connecting the elasticwrist strap 630 to the elastic wrist strap mount 621 or 622. Likewise, athird arm 634 and fourth arm 635 may extend from the central portion 631and may include connection hole 638 for third arm 634 and anunillustrated connection hole for fourth arm 635 for connecting theelastic wrist strap 630 to the elastic wrist strap mount 621 or 622.When mounting to the palm mounted human-machine operation interface 600,the connection holes 636 and 637 may couple with the connection holes622A and 622B and be secured by any reasonable fastener and theconnection hole 638 and the unillustrated connection hole for the fourtharm 635 may couple with the unillustrated connection holes of the firstelastic wrist strap mount 621 and be secured by any reasonable fasteneras shown in FIG. 6B.

As shown in FIGS. 6A and 6B, the palm mounted human-machine operationinterface 600 may include a gripper interface 650 that is configured tocouple one or more of the robotic grippers or hands 400 to the palmmounted human-machine operation interface 600. The gripper interface 650may include an extension portion 651 that extends from the interfacebody 610 and a mount plate 652 that is connected to the extensionportion 651. In some embodiments, the extension portion 651 and mountplate 652 are integral with the interface body 610 and in otherembodiments these components are connected to the interface body 610using any reasonable connection mechanism. The mount plate 652 includesconnection holes 653 and 654 on one side of the mount plate andunillustrated corresponding connection holes on the other side of themount plate.

The gripper interface 650 may also include a coupling member 655 that isconfigured to be the coupling interface with the robotic grippers orhands 400. The coupling member 655 may include a first surface 656 thatconnects with a coupling interface of the robotic grippers or hands 400.A second surface 657 that is on an opposite side of the coupling member655 includes connection holes that correspond to and couple with theconnection holes of the mount plate 652 using any reasonable fastener tosecure the coupling member 655 to the mount plate 652. It will beappreciated that there may be different sizes of the coupling member 655for various different types of the robotic grippers or hands 400.Accordingly, the coupling member 655 may be changed as needed toaccommodate the various different types of the robotic grippers or hands400 such as those shown in FIG. 4 .

As shown in FIGS. 6A and 6B, the palm mounted human-machine operationinterface 600 may include a first camera mount 660 and a second cameramount 665 that are configured to mount a camera 670 and 675,respectively. The camera mount 660 may include a mounting surface 661that includes a connection hole 662A and a connection hole 662B. Themounting surface 661 contacts an underside of the interface body 610 sothat the connection holes 662A and 662B align with connection holes 613Aand 613B of the interface body 610. As shown in FIG. 6B, a fastener suchas a screw is inserted into the connection holes 662A and 613A and theconnection holes 662B and 613B to secure the camera mount 660 to theinterface body 610.

The camera mount 660 also includes connection holes 663 and 664 on alower side of the camera mount. The camera 670 includes mounting surface671 that includes connection holes 672 and 673. The mounting surface 671contacts the lower side of the camera mount 660 so that the connectionholes 663 and 664 align with the connection holes 672 and 673,respectively. As shown in FIG. 6B, a fastener such as a screw isinserted into the connection holes 663 and 672 and the connection holes664 and 673 to secure the camera 670 to the camera mount 660.

The camera mount 665 may include a mounting surface 666 that includes aconnection hole 667A and a connection hole 667B. The mounting surface666 contacts an underside of the interface body 610 so that theconnection holes 667A and 667B align with connection hole 614A and anunillustrated connection hole of the interface body 610. As shown inFIG. 6B, a fastener such as a screw is inserted into the connectionholes 667A and 614A to secure the camera mount 665 to the interface body610. Although not illustrated, a fastener such as a screw is insertedinto the connection holes 667B and the unillustrated connection hole ofthe interface body 610 to also secure the camera mount 665 to theinterface body 610.

The camera mount 665 also includes connection holes 668 and 669 on alower side of the camera mount. The camera 675 incudes mounting surface676 that includes connection holes 677 and 678. The mounting surface 676contacts the lower side of the camera mount 665 so that the connectionholes 668 and 669 align with the connection holes 677 and 678,respectively. As shown in FIG. 6B, a fastener such as a screw isinserted into the connection holes 668 and 677 to secure the camera 675to the camera mount 665. Although not illustrated, a fastener such as ascrew is inserted into the connection holes 664 and 673 to also securethe camera 675 to the camera mount 665.

As shown in FIGS. 6A and 6B, the palm mounted human-machine operationinterface 600 may include a potentiometer interface 680 that isconfigured to mount one or more linear potentiometers that are used tocontrol the operation of the robotic grippers or hands 400. In theembodiment, the potentiometer interface 680 includes sliders 681, 682,683, and 684 that are configured to control a respective linearpotentiometer 686, 687, 688, and 689. As shown in FIGS. 6A and 6B, theinterface body 610 includes elongated openings 615A, 615B, 615C, and615D that allow each of the sliders 681-684 to slide when moved by theuser.

A potentiometer mount 685 is configured to mount the linearpotentiometers 686-689 to the interface body 610. The potentiometermount 685 includes elongated openings 685A, 685B, 685C, and 685D thatcorrespond to the elongated openings 615A-615D. Each of the linearpotentiometers 686-689 includes an opening on a top side for receivingthe sliders 681-684. Thus, each of the linear potentiometers 681-684 ismounted to the potentiometer mount 685 using any reasonable fastenersuch as a screw so that its respective opening is aligned with one ofthe elongated openings 685A, 685B, 685C, and 685D. The potentiometermount 685 is mounted to the interface body 610 using any reasonablefastener such as a screw so that the elongated openings 685A, 685B,685C, and 685D align with one of the elongated openings 615A-615D. Inthis way, each of the sliders 681-684 may be moved by a different fingerof the user, thus allowing for simultaneous control of differentpotentiometers. This action can be seen in FIG. 5 .

As shown in FIG. 6B, an electronics circuit board 690 may be mounted toa bottom side of the interface body 610 using any reasonable mountingmechanism. It will be appreciated that the electronics circuit board 690may include various types of electronics such as a computing device,control circuitry, and the like that can be used to control the variouselements of the palm mounted human-machine operation interface 600 andthe robotic gripper or hand 400 that is attached to the human-machineoperation interface 600. In addition, the various electronics can beused to collect data about the usage of the palm mounted human-machineoperation interface 600 and the connected robotic gripper or hand 400.In some embodiments, the electronics circuit board 690 may be protectedby a heat shield.

Although not illustrated in FIGS. 6A and 6B, in some embodiments thepalm mounted human-machine operation interface 600 may include abi-directional joystick and push button that are mounted on theinterface body 610. The joysticks are installed in the two flangescoming out of the interface body 610. The two small joystick mountsallow the joysticks to be installed with the sticking part pointing upand out of the holes shown in FIGS. 6A and 6B. This allows the user toslide their hands under the flanges when using the slider 681-684 aswell as to control the joysticks by moving the fingers above theflanges. The bi-directional joystick and push button allow for twocontinuous control inputs to the electronics of the electronics circuitboard 690. This can be useful when the device that is being controlledhas coupled actuators that combine to provide motion to a singlecomponent. The bi-directional joystick and push button can be operatedby the user's thumb while the user's four fingers are operating thesliders 681-684 as previously described.

FIG. 5 shows that in some embodiments, an arm support 601 may be coupledto the palm mounted human-machine operation interface 600 to providefurther support to the user. In such embodiments, the arm support 601may be used to provide mounting space for one or more of powerelectronics, external data storage, step-up converters for added powerfor the various robotic grippers or hands 400, and an on-board computer602.

Without prejudice to the invention's principle, the details ofconstruction and the embodiments may vary, even significantly, regardingwhat has been illustrated purely by way of non-limiting example, withoutdeparting from the scope of the disclosure, as this is defined by theannexed claims.

Not necessarily all such objects or advantages may be achieved under anembodiment of the disclosure. Those skilled in the art will recognizethat the disclosure may be embodied or carried out to achieve oroptimize one advantage or group of advantages as taught withoutachieving other objects or advantages as taught or suggested.

The skilled artisan will recognize the interchangeability of variouscomponents from different embodiments described. Besides the variationsdescribed, other known equivalents for each feature can be mixed andmatched by one of ordinary skill in this art to construct the variouscomponents under principles of the present disclosure. Therefore, theembodiments described may be adapted to systems for any suitable device.

Although various embodiments of human-machine interfaces have beendisclosed in certain preferred embodiments and examples, it, therefore,will be understood by those skilled in the art that the presentdisclosure extends beyond the disclosed embodiments to other alternativeembodiments and/or uses of the system and obvious modifications andequivalents. It is intended that the scope of the present systemdisclosed should not be limited by the disclosed embodiments describedabove but should be determined only by a fair reading of the claims thatfollow.

What is claimed is:
 1. A data collection system that performs datacollection of human-driven robot actions for robot learning, the datacollection system comprising: a wearable computation subsystem that isconfigured to be worn by a human data collector and further configuredto control a data collection process by providing one or moreinstructions to and receiving feedback from one or more other subsystemsof the data collection system; a human-machine operation interfacesubsystem that is configured to be worn by the human data collector andfurther configured to allow the data collector to use the human-machineoperation interface to operate an attached robotic gripper to performone or more actions; a user interface subsystem that is configured toreceive instructions from the wearable computation subsystem thatinstructs the human data collector to perform the one or more actionsusing the human-machine operation interface subsystem and that isfurther configured to provide feedback related to the one or moreactions to the wearable computation subsystem; a visual sensingsubsystem that comprises one or more cameras that are configured tocollect raw visual data related to a movement of the robotic gripperwhile performing the one or more actions; and a data collectionsubsystem that is configured to receive collected data related to theone or more actions from one or more of the other subsystem of the datacollection system, wherein the human-machine operation interfacesubsystem is a forearm-mounted human-machine operation interface, theforearm-mounted human-machine operation interface comprising: a forearmstabilizer platform configured to attach to the data collector'sforearm; a gripper support arm having a first end coupled to an end ofthe forearm stabilizer platform; a gripper coupling member coupled to asecond end of the gripper support arm, the gripper coupling member beingconfigured to couple the robotic gripper to the forearm mountedhuman-machine operation interface so that the data collector can operatethe robotic gripper; and a grip handle that is coupled to the grippersupport arm, the grip handle having mounted thereon at least one inputinterface, the at least one input interface being configured to receiveinput from the data collector that is configured to control an operationof the robotic gripper when coupled to the forearm mounted human-machineinterface.
 2. The data collection system of claim 1, wherein thewearable computation subsystem is configured to perform at least thefollowing four functions: 1) real-time synchronization of multiple dataresources, 2) data processing, 3) creating data visualization, and 4)communication with the user interface subsystem.
 3. The data collectionsystem of claim 1, wherein the human-machine operation interfacesubsystem is coupled to a wearable rig vest that is configured to beworn by the data collector.
 4. The data collection system of claim 3,wherein the wearable computation system is configured to be attached toa back side of the wearable rig vest.
 5. The data collection system ofclaim 1, wherein the human-machine operation interface subsystemcomprises one or more sensors that are configured to measure themovement of the human-machine operation interface subsystem or therobotic gripper as the one or more actions are performed.
 6. The datacollection system of claim 1, wherein the human-machine operationinterface subsystem comprises one or more sensors that are configured tomeasure one or more control signals that control the human-machineoperation interface subsystem or the robotic gripper as the one or moreactions are performed.
 7. The data collection system of claim 1, whereinat least one camera of the visual sensing subsystem is a bird-viewcamera that is mounted on a wall or a ceiling of a location where thedata collection is being performed.
 8. The data collection system ofclaim 1, wherein the one or more cameras of the visual sensing subsysteminclude one or more of a depth camera, a tracking camera, or an RGBcamera.
 9. The data collection system of claim 1, wherein the rawsensing data comprises pose data that tracks the robotic gripper in 3Dspace as the robotic gripper is moved when performing the one or moreactions.
 10. The data collection system of claim 1, wherein the userinterface subsystem comprises a Virtual Reality/Augmented Reality(VR/AR) device and a voice user interface.
 11. The data collectionsystem of claim 10, wherein the instructions received from the wearablecomputation subsystem comprise visual instructions that are shown to thedata collector via the VR/AR device or audio instructions that providedto the user via a voice user interface.
 12. The data collection systemof claim 11, wherein the visual instructions include one or moreinstructions to perform the one or more actions that are visually shownin the VR/AR device, a bounding box shown in the VR/AR device thatvisually identifies an object to be interacted with using the attachedrobotic gripper, or status information shown in the VR/AR device relatedto the data collection process.
 13. The data collection system of claim10, wherein the feedback from the user interface subsystem comprises oneor more of a hand gesture that is tracked by the VR/AR device, an audiocommand received by the voice user interface, or a scan of a QR code bythe VR/AR device.
 14. A data collection system that performs datacollection of human-driven robot actions for robot learning, the datacollection system comprising: a wearable computation subsystem that isconfigured to be worn by a human data collector and further configuredto control a data collection process by providing one or moreinstructions to and receiving feedback from one or more other subsystemsof the data collection system; a human-machine operation interfacesubsystem that is configured to be worn by the human data collector andfurther configured to allow the data collector to use the human-machineoperation interface to operate an attached robotic gripper to performone or more actions; a user interface subsystem that is configured toreceive instructions from the wearable computation subsystem thatinstructs the human data collector to perform the one or more actionsusing the human-machine operation interface subsystem and that isfurther configured to provide feedback related to the one or moreactions to the wearable computation subsystem; a visual sensingsubsystem that comprises one or more cameras that are configured tocollect raw visual data related to a movement of the robotic gripperwhile performing the one or more actions; and a data collectionsubsystem that is configured to receive collected data related to theone or more actions from one or more of the other subsystem of the datacollection system, wherein the human-machine operation interfacesubsystem is a palm-mounted human-machine operation interface, thepalm-mounted human-machine operation interface comprising: an interfacebody; a palm support coupled to the interface body; a gripper couplingmember coupled to the interface body, the gripper coupling member beingconfigured to couple the robotic gripper to the palm mountedhuman-machine operation interface so that the data collector can operatethe coupled robotic gripper; and at least one input interface beingconfigured to receive input from the data collector that is configuredto control an operation of the robotic gripper when coupled to the palmmounted human-machine interface.
 15. The data collection system of claim14, wherein the wearable computation subsystem is configured to performat least the following four functions: 1) real-time synchronization ofmultiple data resources, 2) data processing, 3) creating datavisualization, and 4) communication with the user interface subsystem.16. The data collection system of claim 14, wherein the human-machineoperation interface subsystem is coupled to a wearable rig vest that isconfigured to be worn by the data collector.
 17. The data collectionsystem of claim 14, wherein the human-machine operation interfacesubsystem comprises one or more sensors that are configured to measurethe movement of the human-machine operation interface subsystem or therobotic gripper as the one or more actions are performed.
 18. The datacollection system of claim 14, wherein the human-machine operationinterface subsystem comprises one or more sensors that are configured tomeasure one or more control signals that control the human-machineoperation interface subsystem or the robotic gripper as the one or moreactions are performed.
 19. The data collection system of claim 14,wherein at least one camera of the visual sensing subsystem is abird-view camera that is mounted on a wall or a ceiling of a locationwhere the data collection is being performed.
 20. The data collectionsystem of claim 14, wherein the one or more cameras of the visualsensing subsystem include one or more of a depth camera, a trackingcamera, or an RGB camera.
 21. The data collection system of claim 14,wherein the raw sensing data comprises pose data that tracks the roboticgripper in 3D space as the robotic gripper is moved when performing theone or more actions.
 22. The data collection system of claim 14, whereinthe user interface subsystem comprises a Virtual Reality/AugmentedReality (VR/AR) device and a voice user interface.
 23. The datacollection system of claim 22, wherein the instructions received fromthe wearable computation subsystem comprise visual instructions that areshown to the data collector via the VR/AR device or audio instructionsthat provided to the user via a voice user interface.
 24. The datacollection system of claim 23, wherein the visual instructions includeone or more instructions to perform the one or more actions that arevisually shown in the VR/AR device, a bounding box shown in the VR/ARdevice that visually identifies an object to be interacted with usingthe attached robotic gripper, or status information shown in the VR/ARdevice related to the data collection process.
 25. The data collectionsystem of claim 22, wherein the feedback from the user interfacesubsystem comprises one or more of a hand gesture that is tracked by theVR/AR device, an audio command received by the voice user interface, ora scan of a QR code by the VR/AR device.